DXL Attribute: How to pass variable between invocations of script?

Hello,
It's my first post here, so Hello Everyone :)

My question is simple (as usual...):
Is it possible to share some dxl variable (dxl data) between each instance of "attribute dxl" for one attribute?.
For example: to pre-fill skiplist during computation of first "attribute dxl", and use this temporary, pre-filled skiplist for all successive "attribute recalculations" - and clear them out with final call?
Task description:

I've got a task which I solve with "Attribute DXL". The DXL attribute script is applied to each (ca 10 000) object in module DST.
Problem is, that each invocation performs the same time-consuming lookup within module SRC (ca 1000 objs) - looking for one (or none) matching records. This way, almost the same time-consuming procedure is applied to each DST object (for my data 95% of objects lookup returns 'not found' result, but still performing "for each" lookup on each SRC object)

1. This task CANNOT include links (complex baselineing)

Computation time is ca 30 seconds to process DST module per one AttributeDXL (there will be dozen of them in future, performing the same task). Opening such document would take then even 3 - 4 minutes, which is unacceptable. Now we have to live with the 40 seconds penalty...

My question is:
Is there possibility to "prepare" a skiplist and pre-fill it with the LOOK UP TABLE values and use it instead of browsing other module in each record of DST module, in other words: is it possible to make a shared variable between all "attribute dxl" invocations?
Or, do I have to do this with external dll...? Shall I do triggered "on open"?...

I know there are other solutions, like making a 'trigger' on opening document and do the recalculation then.

Kind regards,
Piotr Wierzchowski
piotrwie - Fri Jan 21 05:59:05 EST 2011

Re: DXL Attribute: How to pass variable between invocations of script?
Mathias Mamsch - Fri Jan 21 11:27:06 EST 2011

Yes you can do this, but there is a problem. You need to ensure, that the data is still valid on the subsequent calls to the calculation, and the even bigger problem is changing the cached data in subsequent calls to dxl programs.

A little background on DOORS memory management: Each DXL program (yes even each attribute dxl or layout dxl calculation) lives in its own DXL context. The DXL context has a list of allocated variables (e.g. if your DXL script allocates a buffer, the memory consumed by it will be associated to the DXL context). If the DXL program ends, then all memory will be released, that is every string declared, every literal of your program, everything. So normally every precached value of another script will be lost during the next script running.

There are two ways to end a program and keep the memory alive: a) 'show' a GUI b) define a dynamic trigger. A dynamic trigger bound to a module, will release its memory when the module closes (which is probably what you want) or when the trigger is deleted, so by defining for example an 'pre close' trigger in the program that does the calculations you can make sure that the memory will stay alive. The next question is how to pass the memory to another program. For this you need to store the memory address somewhere, where the other program can access it. I changed the name of the dynamic trigger (using a bad hack) to contain the memory adress of the data, which is neat, because this also allows you to identify the trigger (to delete it later from another script).

The next alternative is using the global context with 'evalTop_'. This is much easier then the first approach, but has a couple of drawbacks:

a) You need to define the global variable that stores your data exactly once. The variable will be available until the DOORS client ends.

b) evalTop_ will only be executed after the program calling it ends, so you would need to invoke the global level script from another script (like an "post open module" trigger, but this will again introduce the open-penalty).

So which one shall it be? :-)

Regards, Mathias


Mathias Mamsch, IT-QBase GmbH, Consultant for Requirement Engineering and D00RS

Re: DXL Attribute: How to pass variable between invocations of script?
llandale - Fri Jan 21 15:46:34 EST 2011

Mathias' response seems beyond the capabilities of a first time poster, although he indeed answered the question well.

I would just have an on-demand script that did the evaluation and then set some string attribute with the results for all objects, and schedule this script to run every night. Make the script available for folks who cannot stand 3 hour old data.

But while I'm thinking about it, let me think out loud and let someone comment about this silly notion.

You could define your Text attrDXL to apply to both Objects and the Module. The DXL itself would get the obj."Absolute Number" and if that value were null, then it would know that it was running at the module context, otherwise it knows its running at the object context.

When running at the Module context, the script would do your very clever evalyuation of the other module, then loop through all objects in this module and calculate the result for that object. All the object results are stored at the module context (very large results).

When running at the Object context, the object would query the Module context, search for this objects' results, and display it. The format for each object may be

identifier tab results eol

It should be the case that the first object to need the attr DXL would try to retrieve the module value, which should initiate its 30 seconds running, but each additional object would just retrieve that value and it would not be re-calculated.

You would need a good string searcher, Regular Expressions will be slow here. I suspect, however, the Buffer 'contains' would work for identifiers.

The following attr DXL, when defined for Objects and Module, proves the concept that the code can determine when its running at the module context. The Module level attr value is "Module null".

// dxlModuleLevel
string    AbsNo = obj."Absolute Number" ""
if (AbsNo == "0")  //
then obj.attrDXLName = "Module 0"
elseif(null AbsNo)      //
then obj.attrDXLName = "Module null"
else obj.attrDXLName = AbsNo


So perhaps your code would look like this:

 

string AbsNo = obj."Absolute Number" ""
Buffer    bufResults = create()
string    Results
 
if (null AbsNo)
{          // Running at module context
    open the other module
    create your other module data structures
    evaluation the other module
    close the other module
    for o in entire (current Module) do
    {  Res = results for this object o
       bufResults += identifier(o)
       bufResults += "\t"
       bufResults += Res
       bufResults += "\t" charof(1) charof(2) charof(3) \n"  // End of line indicator, searched for below
    }
    Results = tempStringOf(bufResults)
    delete your other module data structures
}    // end module context process
else
{    // object context procesing:
   bufResults = mod.attrDXLName
   int PositionID = contains(bufResults, identifier(obj), 0)
   if (PositionID < 0) no results??
   int PositionRes = PositionID + length(idenifier(obj) +1 // +1 skips the tab
   int PositionEnd = contains(bufResults, charof(1) charof(2) charof(3) "", PositionRes)
   Results = bufResults[PositionRes:PositionEnd-1]
}
obj.attrDXLName = Results
delete(bufResults)

 

 

  • Louie

 

Re: DXL Attribute: How to pass variable between invocations of script?
piotrwie - Wed Jan 26 06:24:57 EST 2011

Mathias Mamsch - Fri Jan 21 11:27:06 EST 2011
Yes you can do this, but there is a problem. You need to ensure, that the data is still valid on the subsequent calls to the calculation, and the even bigger problem is changing the cached data in subsequent calls to dxl programs.

A little background on DOORS memory management: Each DXL program (yes even each attribute dxl or layout dxl calculation) lives in its own DXL context. The DXL context has a list of allocated variables (e.g. if your DXL script allocates a buffer, the memory consumed by it will be associated to the DXL context). If the DXL program ends, then all memory will be released, that is every string declared, every literal of your program, everything. So normally every precached value of another script will be lost during the next script running.

There are two ways to end a program and keep the memory alive: a) 'show' a GUI b) define a dynamic trigger. A dynamic trigger bound to a module, will release its memory when the module closes (which is probably what you want) or when the trigger is deleted, so by defining for example an 'pre close' trigger in the program that does the calculations you can make sure that the memory will stay alive. The next question is how to pass the memory to another program. For this you need to store the memory address somewhere, where the other program can access it. I changed the name of the dynamic trigger (using a bad hack) to contain the memory adress of the data, which is neat, because this also allows you to identify the trigger (to delete it later from another script).

The next alternative is using the global context with 'evalTop_'. This is much easier then the first approach, but has a couple of drawbacks:

a) You need to define the global variable that stores your data exactly once. The variable will be available until the DOORS client ends.

b) evalTop_ will only be executed after the program calling it ends, so you would need to invoke the global level script from another script (like an "post open module" trigger, but this will again introduce the open-penalty).

So which one shall it be? :-)

Regards, Mathias


Mathias Mamsch, IT-QBase GmbH, Consultant for Requirement Engineering and D00RS

Hello Mathias, llandale

Thank you very much for your emails.
I think I'll experiment with all approaches you're describing. I've still got some time (2 months?) to evaluate which one suits our solution better.

P.S.

I'm not a novice in scripted data processing, nor programming, but I'm more and more amazed how deep is the DOORS knowledge among people on this forum, and how almost non-existing is the actual (official) documentation on it...
I'll try to do it first using shared memory (according to guidelines from Mathias) - this is the fastest solution here. I want simply to share a large single skiplist and allow all dxlAttributes to perform lookups according to its contents.

The data unfortunately must be updated frequently...

Regards,
Piotr

Re: DXL Attribute: How to pass variable between invocations of script?
piotrwie - Wed Jan 26 06:26:15 EST 2011

llandale - Fri Jan 21 15:46:34 EST 2011

Mathias' response seems beyond the capabilities of a first time poster, although he indeed answered the question well.

I would just have an on-demand script that did the evaluation and then set some string attribute with the results for all objects, and schedule this script to run every night. Make the script available for folks who cannot stand 3 hour old data.

But while I'm thinking about it, let me think out loud and let someone comment about this silly notion.

You could define your Text attrDXL to apply to both Objects and the Module. The DXL itself would get the obj."Absolute Number" and if that value were null, then it would know that it was running at the module context, otherwise it knows its running at the object context.

When running at the Module context, the script would do your very clever evalyuation of the other module, then loop through all objects in this module and calculate the result for that object. All the object results are stored at the module context (very large results).

When running at the Object context, the object would query the Module context, search for this objects' results, and display it. The format for each object may be

identifier tab results eol

It should be the case that the first object to need the attr DXL would try to retrieve the module value, which should initiate its 30 seconds running, but each additional object would just retrieve that value and it would not be re-calculated.

You would need a good string searcher, Regular Expressions will be slow here. I suspect, however, the Buffer 'contains' would work for identifiers.

The following attr DXL, when defined for Objects and Module, proves the concept that the code can determine when its running at the module context. The Module level attr value is "Module null".

// dxlModuleLevel
string    AbsNo = obj."Absolute Number" ""
if (AbsNo == "0")  //
then obj.attrDXLName = "Module 0"
elseif(null AbsNo)      //
then obj.attrDXLName = "Module null"
else obj.attrDXLName = AbsNo


So perhaps your code would look like this:

 

string AbsNo = obj."Absolute Number" ""
Buffer    bufResults = create()
string    Results
 
if (null AbsNo)
{          // Running at module context
    open the other module
    create your other module data structures
    evaluation the other module
    close the other module
    for o in entire (current Module) do
    {  Res = results for this object o
       bufResults += identifier(o)
       bufResults += "\t"
       bufResults += Res
       bufResults += "\t" charof(1) charof(2) charof(3) \n"  // End of line indicator, searched for below
    }
    Results = tempStringOf(bufResults)
    delete your other module data structures
}    // end module context process
else
{    // object context procesing:
   bufResults = mod.attrDXLName
   int PositionID = contains(bufResults, identifier(obj), 0)
   if (PositionID < 0) no results??
   int PositionRes = PositionID + length(idenifier(obj) +1 // +1 skips the tab
   int PositionEnd = contains(bufResults, charof(1) charof(2) charof(3) "", PositionRes)
   Results = bufResults[PositionRes:PositionEnd-1]
}
obj.attrDXLName = Results
delete(bufResults)

 

 

  • Louie

 

Thanks llandale,

please see my reply to Mathias - I thought it would appear at the bottom, so it was addressed to both of you.

Piotr

Re: DXL Attribute: How to pass variable between invocations of script?
PDU - Wed Jan 26 08:40:42 EST 2011

piotrwie - Wed Jan 26 06:26:15 EST 2011
Thanks llandale,

please see my reply to Mathias - I thought it would appear at the bottom, so it was addressed to both of you.

Piotr

Hello Piotr,

i think to an other solution : save data of first script into a file.

Perhaps not very fast, but easy.

Pierre

Re: DXL Attribute: How to pass variable between invocations of script?
Mathias Mamsch - Wed Jan 26 11:55:43 EST 2011

piotrwie - Wed Jan 26 06:24:57 EST 2011
Hello Mathias, llandale

Thank you very much for your emails.
I think I'll experiment with all approaches you're describing. I've still got some time (2 months?) to evaluate which one suits our solution better.

P.S.

I'm not a novice in scripted data processing, nor programming, but I'm more and more amazed how deep is the DOORS knowledge among people on this forum, and how almost non-existing is the actual (official) documentation on it...
I'll try to do it first using shared memory (according to guidelines from Mathias) - this is the fastest solution here. I want simply to share a large single skiplist and allow all dxlAttributes to perform lookups according to its contents.

The data unfortunately must be updated frequently...

Regards,
Piotr

I would also suggest you to use a simpler approach, like a file (once they are cached, access is pretty quick) or Louies clever Module/Object DXL attribute solution.

Especially updating data in a global memory cache is pretty hard. The reason is, that updating data that resides in another DXL context is most likely a bad idea. When your DXL program ends, then all data (including for example all strings) in the programs DXL context is going to be invalidated in memory, ready to be overwritten by the next memory allocation. So therefore assume you have stored your cached data in some DXL context, either one you keep alive with a trigger or the global context, then you want to have a script XY that you start from the menu update the contents of your cache. So the XY script will get the address of the cache and write some data to it. Unfortunately as soon as the XY script ends all data that it allocated will be invalidated. So for example putting an entry to a skip list will most likely not be stable from another context, because it will allocate memory for the skip entry, that will be freed when the XY script ends. What will work is updating an existing entry in a skip list, because this will do no allocation. BUT: don't write a string to the skip list, because again the string will be allocated in the context of XY and freed afterwards.

I guess you can see the problem. Switching to the global context is pretty easy using the evalTop_ function, but this will only run after the XY script has ended. I experimented with running evalTop_ from DOORS over COM (i.e. the DOORS client invoking itself to run a script over COM), which indeed works, but assumes that COM is installed properly and even more critically only one DOORS client instance is started (since COM will always invoke the client started first).

Therefore I thought about the possibility of "switching" to the DXL context in which the cache resides, then do the update there. Unfortunately there is no "SwitchToContext" perm in DXL :-) So my idea was to install a "pre close" trigger that will keep the memory alive. To switch to the trigger context I just need to invoke "close" on the module, then the trigger will evaluate in its own context (hopefully, I am not 100% sure about that, but it seems to work!). Then this trigger will again invoke a callback function that was defined in the XY script. I got that working but it is really hard to get right, I wanted to attach the code, but I cannot get an example running in this moment.

So again, I would suggest for a simple mapping like you described, to use another approach, like a file containing an index part, which will let you jump quickly to the index that you need. Handling global data in DOORS is really a subject where you can quickly waste a lot of trial and crash time. And you should only go this way, if you have very complex data structures, that will take a lot of time when marshalling to a string representation.

Regards, Mathias


Mathias Mamsch, IT-QBase GmbH, Consultant for Requirement Engineering and D00RS